A Performance Measure for Classification with Ambiguous Data

Real world data can be difficult to classify due to overlapping classes of ambiguous data. One solution to this problem is to leave out data before classifying, while another solution is to first classify the data and then prune those results which are ambiguous. However, a problem exists in determining which data are ambiguous. In this paper we propose a performance criteria which gives a precise basis for characterizing the performance of any classifier applied to ambiguous data. Further, we demonstrate that there is an optimal region of withholding classifications which depends on the performance criteria. We test our method on some benchmark classification problems to show the effectiveness of the approach.